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Overview 


•  Traditional  metrics  of  program  performance  have  been  cost, 
schedule  and  system  requirements,  relative  to  the  APB 
baselines  set  at  a  previous  point  in  time 

-  Some  programs  perform  well,  but  most  do  not 

•  What  is  the  impact  of  defense  acquisition  regulation  or 
reforms? 

-  Previously  modeled  as  the  correlation  and  direction  of  change  for 
a  key  performance  metric  for  a  program  in  the  years  following  the 
introduction  of  the  oversight  or  reform  change 

-  Provides  absolutely  no  insight  on  the  actual  mechanism  behind 
the  implied  causality 

•  Hypothetical:  What  went  on  (that  was  internal  to  programs)  that  caused  cost 
growth  to  slow  after  FASA  in  1994,  or  to  show  no  effect  from  Packard 
Commission  reforms  in  1987  ? 
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Overview 


or 


We  take  a  new  approach  here,  where  a  mechanism  is  already  in 
place  to  create  program  changes 
-  DAES  reviews  and  requirements  for  corrective  actions 

•  Recurring  program  assessment  ratings 

•  Defined  corrective  steps  and  timeline 

•  Follow  up  occurs  in  the  next  DAES  review 

The  following  models  will  determine  if  regulation,  oversight  and 
a  variety  of  other  factors  produce  desired  or  expected  results 
on  the  ratings  process  -  and  drive  the  corrective  steps  that 
have  to  be  taken  by  program  management  before  the  next 
review. 


or 


NUCLEAR 

SECURITY 


4 


©  2014  by  the  University  of  Tennessee  Institute  for  Nuclear  Security 


Method 


9  Like  a  report  card,  DAES  program  assessment  reviews 

carry  consequences  for  defense  programs:* 

-  Green 

-  Yellow 

-  Red 

9  If  regulatory  and  oversight  actions,  or  cost  performance,  etc 
support  a  Green  rated  program  continuing  to  be  Green  rated, 
this  is  precisely  the  desired  result. 

9  On  the  other  hand,  if  regulatory  or  oversight  actions  drive  a 
Green  rated  program  toward  Yellow  or  Red,  then  the  corrective 
mechanism  comes  into  play  and  corrections  must  be  made  - 
and  the  PARCA  /  DAES  Deskbook  spells  out  the  requirements 
and  time  frame. 


*  Department  of  Defense  (2012).  Defense  Acquisition  Executive  Summary  Assessment  Deskbook.  Washington,  DC.  OUSD/AT&L 
Performance  Assessment  and  Root  Cause  Analysis.  Version  14.  December. 
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Method 


•  Ordered  logit  models 

-  Efficiently  use  the  information  contained  in  the  ranked  values  of 
dependent  (response)  variable,  i.e.  Red  «  Yellow  «  Green 

-  Response  (dependent )  variable  is  recoded  1,2,3,  and  the 
functional  form  assumes  a  logistical ly  distributed  error  term 

-  Four  models 


•  Focus  is  on  two  DAES  rating  categories  (Cost  and  Schedule)  -  from 
two  points  of  view,  Program  Management  and  OSD 


•  Exploratory  work  also  modeled  binary  recodes  of  assessment 
rating  variables  to  understand  contrasts,  i.e.,  program  was 
either  green  (=1)  in  time  period  j  or  it  was  not  (=0) 

-  Several  variants  on  same  principle 

-  Points  the  way  for  future  work  to  refine  our  understanding  of 
impact  of  breach  run-in  and  breach  recovery,  funding  instability, 
rebaselining,  cancellation 
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Data 


Programs  have  been 
DAES  reviewed  and  rated 
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Data 


•  Time  series,  cross  sectional  panel,  2001-2013 

-  Program  Manager  (PM)  assessment  ratings  for  Cost  and  Schedule 
categories 

-  OSD  DAES  review  team  assessment  ratings  for  Cost  and  Schedule 
categories 

•  Program  Selection: 

-  57  major  defense  programs  included  in  sample 

-  Minimum  3,  maximum  13  years.  Average  8-9  years. 

-  19  (+/-)  MDAPs  sampled  per  service 

9  Regulatory  data  methodology  followed  the  Krieger  and  Pritchard* 
method  for  counting  acquisition  regulation  measures. 

9  Service  specific  manpower  counts  (GS1102  contracting  officers). 

9  Program  specific  data  from  SARs,  APBs. 


*  John  Kreiger  and  John  Pritchard  (2009)  "Acquisition  Reform  and  the  Golf  Ball."  Contract  Management  49  (12).  18-27.  December. 
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Programs 
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Models  and  Results 


9  Four  models,  described  by  the  dependent  variable  we  wish  to 
explain 

-  Program  Manager  team  Cost  assessment  of  the  program 

-  OSD  DAES  Review  team  Cost  assessment  of  the  program 

-  Program  Manager  team  Schedule  assessment  of  the  program 

-  OSD  DAES  Review  team  Schedule  assessment  of  the  program 

9  Reading  the  table  on  the  next  slide: 

-  Negative  sign  on  coefficient  estimates  indicates  that  an  increase  in 
the  explanatory  variable  correlates  with  worse  ratings  (yellow  or 
red);  a  positive  sign  predicts  better  ratings 

-  Statistical  significance  is  noted  by  asterisks  -  the  more  asterisks, 
the  higher  the  statistical  significance 
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Interpretation 


•  Previous  period  rating  is  a  factor,  as  expected,  but  the  models 
clearly  tell  us  that  there  is  a  bigger  story  in  ratings  program 
performance 

•  Regulation  (NDAA)  as  we  tested  it  here,  was  significant  -  but 
not  DARS  notifications. 

-  Better  approach  would  use  qualitative  distinctions 

on  type  and  specificity  of  regulation  ( 

•  Oversight  matters 

-  GAO  mention  of  a  program  in  “ Assessments  of  Selected 
Weapons  Programs...” 

-  ACAT  ID  and  level  of  program  decisional  authority  matters 


r\ 
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Interpretation 


or 


System  type 

-  Clear  relationship  of  some  system  types  with  poorly  rated 
programs  (vehicles,  UAV/UAS)  or  better  rated  programs 
(munitions,  missiles) 

Supports  and  validates  the  significance  of  cost  growth  and 
schedule  slip  as  concerns  to  the  assessment  teams 

Obvious  independence  of  viewpoints  not  only  between  Cost 
and  Schedule  reviews,  but  also,  between  the  PM  team  and  the 
OSD  team  reviewers  (this  is  a  good  thing) 
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Conclusion 


•  Models  suggest  a  better  way  to  evaluate  if  process  or  oversight 
reform  works 

-  There  is  a  closed  loop  feedback  mechanism  in  place,  with  periodic 
objective  review  by  experienced  review  teams. 

-  The  DAES  review  assessment  rating  process  holds  promise  for 
suggesting  why  a  reform  works  -  or  doesn’t 

9  Oversight  does  matter  -  and  apparently,  rather  than  being  a 
burden,  may  efficiently  drive  program  improvements 

9  Econometric  modeling  lesson  is  that  this  is  opportunity  for 
better  understanding  of  cause,  effect,  and  explanation  of 
defense  acquisition  improvements  like  Better  Buying  Power 

f 
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Future  Direction 


9  Program  database  continues  to  grow  after  we  closed  the  data 
set  for  this  study 

9  Expanding  sample  allows  for  work  on 

-  Program  phase  (EMD,  Production);  cancelled  and  restructured 
programs;  Nunn-McCurdy  process  on  program  outcomes 

-  Frequency  of  rebaselining  and  program  quality 

-  Assessment  ratings  on  Funding  and  System  Requirements 
(“Performance  ”) 

9  Fix  shortcomings  on  regulatory,  oversight  and  manpower  data 
we  were  unable  to  obtain 

9  Add  program  specific  and  period  event  data 

9  Expand  on  testing  and  insights  from  other  modeling  methods 
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